{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UK research networks with HoloViews+Bokeh+Datashader\n",
    "\n",
    "[Datashader](http://datashader.readthedocs.org) makes it possible to plot very large datasets in a web browser, while [Bokeh](http://bokeh.pydata.org) makes those plots interactive, and [HoloViews](http://holoviews.org) provides a convenient interface for building these plots.\n",
    "Here, let's use these three programs to visualize an example dataset of 600,000 collaborations between 15000 UK research institutions, previously laid out using a force-directed algorithm by [Ian Calvert](https://www.digital-science.com/people/ian-calvert).\n",
    "\n",
    "First, we'll import the packages we are using and set up some defaults."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import holoviews as hv\n",
    "from holoviews import opts\n",
    "import fastparquet as fp\n",
    "\n",
    "from colorcet import fire\n",
    "from datashader.bundling import directly_connect_edges, hammer_bundle\n",
    "\n",
    "from holoviews.operation.datashader import datashade, dynspread\n",
    "from holoviews.operation import decimate\n",
    "\n",
    "from dask.distributed import Client\n",
    "client = Client()\n",
    "\n",
    "hv.notebook_extension('bokeh','matplotlib')\n",
    "\n",
    "decimate.max_samples=20000\n",
    "dynspread.threshold=0.01\n",
    "datashade.cmap=fire[40:]\n",
    "sz = dict(width=150,height=150)\n",
    "\n",
    "opts.defaults(\n",
    "    opts.RGB(width=400, height=400, xaxis=None, yaxis=None, show_grid=False, bgcolor=\"black\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The files are stored in the efficient Parquet format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r_nodes_file = '../data/calvert_uk_research2017_nodes.snappy.parq'\n",
    "r_edges_file = '../data/calvert_uk_research2017_edges.snappy.parq'\n",
    "\n",
    "r_nodes = hv.Points(fp.ParquetFile(r_nodes_file).to_pandas(index='id'), label=\"Nodes\")\n",
    "r_edges = hv.Curve( fp.ParquetFile(r_edges_file).to_pandas(index='id'), label=\"Edges\")\n",
    "len(r_nodes),len(r_edges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can render each collaboration as a single-line direct connection, but the result is a dense tangle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time r_direct = hv.Curve(directly_connect_edges(r_nodes.data, r_edges.data),label=\"Direct\")\n",
    "dynspread(datashade(r_nodes,cmap=[\"cyan\"])) + datashade(r_direct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Detailed substructure of this graph becomes visible after bundling edges using a variant of [Hurter, Ersoy, & Telea (ECV-2012)](http://www.cs.rug.nl/~alext/PAPERS/EuroVis12/kdeeb.pdf), which takes several minutes even using multiple cores with [Dask](https://dask.pydata.org):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time r_bundled = hv.Curve(hammer_bundle(r_nodes.data, r_edges.data),label=\"Bundled\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dynspread(datashade(r_nodes,cmap=[\"cyan\"])) + datashade(r_bundled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zooming into these plots reveals interesting patterns (if you are running a live Python server), but immediately one then wants to ask what the various groupings of nodes might represent. With a small number of nodes or a small number of categories one could color-code the dots (using datashader's categorical color coding support), but here we just have thousands of indistinguishable dots. Instead, let's use hover information so the viewer can at least see the identity of each node on inspection.  \n",
    "\n",
    "To do that, we'll first need to pull in something useful to hover, so let's load the names of each institution in the researcher list and merge that with our existing layout data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "node_names = pd.read_csv(\"../data/calvert_uk_research2017_nodes.csv\", index_col=\"node_id\", usecols=[\"node_id\",\"name\"])\n",
    "node_names = node_names.rename(columns={\"name\": \"Institution\"})\n",
    "node_names\n",
    "\n",
    "r_nodes_named = pd.merge(r_nodes.data, node_names, left_index=True, right_index=True)\n",
    "r_nodes_named.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now overlay a set of points on top of the datashaded edges, which will provide hover information for each node.  Here, the entire set of 15000 nodes would be reasonably feasible to plot, but to show how to work with larger datasets we wrap the `hv.Points()` call with `decimate` so that only a finite subset of the points will be shown at any one time. If a node of interest is not visible in a particular zoom, then you can simply zoom in on that region; at some point the number of visible points will be below the specified decimate limit and the required point should be revealed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "edges = datashade(r_bundled, width=900, height=650)\n",
    "points = decimate( hv.Points(r_nodes_named),max_samples=10000)\n",
    "(edges * points).opts(\n",
    "    opts.Points(color=\"cyan\", tools=[\"hover\"], width=900, height=650),\n",
    "    opts.Overlay(width=900, height=650))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you click around and hover, you should see interesting groups of nodes, and can then set up further interactive tools using [HoloViews' stream support](http://holoviews.org/user_guide/Responding_to_Events.html) to reveal aspects relevant to your research interests or questions.\n",
    "\n",
    "As you can see, datashader lets you work with very large graph datasets, though there are a number of decisions to make by trial and error, you do have to be careful when doing computationally expensive operations like edge bundling, and interactive information will only be available for a limited subset of the data at any one time due to data-size limitations of current web browsers."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
